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EXECUTIVE OFFICE OF THE PRESIDENT 
Bureau of the Budget 
Washington, D. C. 20503 


Marcr 29. 1566 


MEMORANDUM FOR: Mr. John Clarke 


I am sending you some copies of a Program Evaluation 
Checklist that we will be using in the Budget Bureau ir 
the coming months. You might want to circulate copies >1f 
the checklist to members of your staff and throughout 

the agency. 


fog bom 


HENRY S. ROWEN 


Assistant Director 


Va 
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committed “to these objectives? What form does the ‘commitment 


wha 
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PROGRAM EVALUATION CHECKLIST 


The following list of questions has been assembled for 
the use both of consumers and producers of key documents in 
the Planning-Programming~Budget ing System. It focuses mainly | 
on the analyses | to_be reflected . _in..the Program ram Memoranda (PM) 


although “some points | relate to _the Program and_ Financial Flans 


(PFP). 
I. OBJECTIVES 


Perhaps the most difficult and important task of analy- 
sis of programs in the public sectior is being clear on ob- 
jectives and the contribution of specific programs to specific 
objectives. 


1. What assertions are made about overall national ob- pr 
jectives? What authority is quoted? The President? The 
Congress? The Agency Head? How specific and concrete is 
the statement of objectives? ‘To what extent is the Government 


take? Presidential statements? Statutes? Agreement with 
Congressional committees? Moral obligations? How much room 
for 1 rpretation is there? For example, is it clear over 

rae eriod the objectives should be met? Does the PM 
transiate genéral, , abstract, vague objectives into opera- 
tionally meandngeul ones? 


2. Is it clear what should be the Federal Government.'s 
responsibility re the cited or proposed objectives? Has the 
alternative of greater contribution by State _ and local agen-. 


cies « or the private sector “been ‘considered? “Should it be a 
Federal al responsibility at all? 


3. Is the relationship between broad national goals 
and specific agency goals clearly stated? Is it reasonabie? 
What policies, legislation and statements of fact are impiicit 
in the objectives? Should they be made explicit? Are they 
right? 


4. Are the agency goals stated in terms which have op- 
erational meaning? Have they “been ‘expressed quantitatively? 
Is there a time- table for accomplishment? | 
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ent parts of ee same agency .¢ or in different agencies been 
made explicit (e.g., the War on Poverty)? Does the analysis 
adequately take into account other efforts to accomplish 

these goals -- efforts by other agencies of the Federal Gov- 
ernment, by State or local governments, or by the private 

sector? Is the impact of the agency's programs on the goals 
of other agencies noted? 


6. Have the agency'd priorities hmong objectives and 
programs been made clear? If more, or fewer, funds are 
available than in the basic case assumed by the agency, is 
it evident what the agency would propose to add or delete 
from its programs? 


7. What is the evidence that the alleged "need" for 
goods and services exists in the sense that people would be 
willing to pay for the service? Given this evidence, does 
the analysis show that the agency's proposed scale of op- 

ILLEGIBeration is best? Or has the agency proposed fulfillment of 
all potential "need" for the service by employing an im- 
plicit price level of zero? On the other hand, has it 
stopped short of fulfilling demand willing to meet the 
proper price? Why? Was a budget constraint the reason 
for this? Can a better budget justification be made if 
this is the case (perhaps as the result of further study) ? 


II. MEASURES OF EFFECTIVENESS 


The private sector (and certain business-like govern- 
mental activities) have profits which serve as a guide to 
performance; most governmental agencies have to develop 
their own specific indicators of performance. This is often 
difficult to do well and errors of various kinds are frequent. 


1. To what extent are the tangible, measurable outputs 
the program used by the agency adequate ‘indicators of the 
extent to which the overall purposes of the program are 
being achieved? If inadequate, what better measures of 
performance might be used? 


2. Are there clearly identified criteria offered by 
which to judge program performance? How adequate do they a, 
seem to be? Are there some alternative criteria that should ea 
be considered? 
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Have outputs been quantified to the extent useful? 
ere quantitative measures of program performance are miss - 


ing ore theses logical, convincing qualitative assessments? 
as the agency gone too far and at ato quantify the — 
unquantifiable? Is it clear where dependence on the quanti- 
tative outputs leaves off and judgment takes over? 


4. Are there close substitutes for these outputs whiz: 
will affect the demand for them? Are these substitutes con- 
sidered in the analysis? 


ws 


III. FINANCIAL DATA 


Whereas it is often very difficult to define and obtain 
good measures of program effectiveness, the cost side is 
usually easier to cope with. And often a better understanc-~ 
ing of costs can be very useful in improving program perform- 
ance. 


1. Does the PM present the costs that are relevant tc 
the issues analyzed? Have systems costs estimates been _ 
developed and presented? How complete and how accurate are 
these estimates? Have any valuable inputs been ignoreci? 
Are all directly related support training and construction 


costs included? “What additional cost analytic efforts are 
gested by the quality of the agency's cost data? 
? 


2. Does the analysis describe the bases for the sost 
imates? Do the estimates look reasonable? 


3. Have sunk costs -- past sacrifices -- been omitted 

as they should be, except where there are significant sal- 
vage values? Have they been considered where they should Le - 
i.e. in the evaluation of the past program? 


4. In systems with multiple products, have only the 
nonavoidable costs been charged to a given program, or 

have common costs been arbitrarily allocated? What is 

the purpose of the allocations? Have popular programs been 
saddled with part of the burden of less attractive programs” 
That is, have costs been shifted from where they should 
properly be charged to where they can most easily be funded” 
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Have costs been shifted from products with easily measured 
output to products with only qualitative outputs? For ex- 
ample, in a multipurpose dam project, are costs overallo- 
cated to recreation and underallocated to irrigation or 
power products? 


5. Is the agency too optimistic in its assumptions 
“ on funds likely to be available for its programs? What 
effect should more modest assumptions have on the conclu- 
sions? 


6. Is there available a record of successive agency cost 
stimates over time of new programs? What does this suggest 
bout the accuracy of current claims? Why shouldn't the 


sn bem 


7. Is there a sensitivity analysis to show the effect ? 


of uncertainty in cost estimates on conclusions? What evi- 
dence is adduced in support of its cost estimates? 


8. Have future benefits and costs been discounted? 
For example, if a proposed project is compared with exist- 
ing ones for which future Savings are expected, are those 
future savings discounted? On the other hand, is the case 
for discounting future outlays and benefits negated by the 
existence of a brief time span for the project, or by un- 
certainties? 


9. Is the chosen discount rate reasonable, i.e., does 
it reflect alternative uses of investible funds? How 
sensitive are conclusions on preferred programs to the 
choice of discount rates? ve 


IV. SOME ADDITIONAL POINTS 


Treatment of Alternatives 

1. How thoroughly have alternative methods for accom- 
plishing the same objectives been analyzed? Are.there 
alternatives involving different—tactics) or\techniques, > 
different modes of production. (more or less capital inten- 
sive), different priorities, different mixes of Federal 
vs. State and local expenditures, different financing 
methods, different legislative requirements? Were differ- s™. 
ent levels for the agency's preferred program studied? Are 


mixtures of systems considered among the alte ives? 
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“¥adically different programs? 


Approved For elease 2005/11/28 : CIA-RDP86M00612%000100030020-9 


eae 


2. Have the costs and effectiveness of the major ai- 
ternatives been properly assessed? Have tradeoffs been 
made between present and proposed programs? Were the cri- 
teria, costing methods and time span the same as for the 
proposed program, i.e., was the comparison fair? 


3. Has adequate account been taken of the need to 
examine programs at the margin? Quite often the issue is 
not whether to have a program at all, but whether to make 
Marginal changes in the program level or composition. 
Does the analysis deal with specific issues in terms of 
the marginal benefits and marginal costs? Does the data 
displayed show the effect of Marginal changes or current 
marginal operations? 


4. Has the presentation of program data in terms of 
rithmetic averages concealed important information about 
he spread in values? If an analysis supports certain cor- 
clusions about a program based on these averages, might 
there nevertheless be important exceptions? What are 
they? Should they be examined? 

a 

2: Are the alternatives analyzedregl’ options for_ 
the agency head? Does he really have a choice? On the 


Other hand, is the¥e Inability or unwillingness to conside~ 


set 


rata otis aS or ee 


6. Have the constraints on the program area beer ex- 


'plicitly treated? What are they? Have ways of modifying 


the constraints been explored? What reasons have been 
given for rejecting alternative objectives or programs? 
Efficiency? Statutory prohibitions? Organizational limi- 
tations? Lack of trained personnel? tack of capital? 
Inadequate technology? 


Models 


1. Is there presented a model or simplified, quanti- 
tative description of some important aspects of the pro- 
gram? If this hasn't been attempted for a broad program 
area, has it been attempted for some important parts? Has 
the analyst failed to structure the important relation~ 
ships explicitly? How adequately does the model seem <o 
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explain observed phenomena? Does it rest on structural re- 
Lationships or is it based entirely on statistical corre- 
lation, i.e., is a theory offered? Does it seem plausible? 


2. Can you suggest specific improvements on the model? 
What factors should be included that have been left out? 


3. Has the analyst engaged in excessive modelism, i.e., 
does his work reflect more interest in the model than in the 
real world? Has he focused on manipulating a computational 
method at the expense of dealing with important factors in 
a sensible way? 


4. Are there severe suboptimizations? Have elements 
that should be regarded as variables by the agency head or 
the President been allowed to vary, or are there restric— 
tions imposed in the process of dividing the analysis into 
lower level problems? : 


Treatment of the Future 


ee rT 


1. Has the proper span of years been considered? 
Does the PFP or the PM go too far, or not far enough, into 
the future? Beyond the immediate period, is there simply 
a mechanical projection of recent trends which fails to 
take adequate account of possible changes in policy, 


technology, costs, or availability of funds? 


2. Has an explicit, time phased, decision strategy 
been presented which allows for options being opened up. 
and foreclosed, research being done, experiments being 
carried out, and data gathered? Alternatively, is the 
program area characterized by a collection of unstructured 
ad hoc programs and decisions without a coherent theory?. 
If the latter, what should be done to improve the situa- 
tion? 


3. Does the PM show a range of possible future 
options as to program objectives and means of achieving 
these objectives? Does it show over what period of time 
in the future it should become possible to choose among 
these options? Does it show what evidence should dictate 
a choice among options? 
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4. Is the proposed program level shown. dependent on 
how well the program performs over the next year or two? 
Is there any discussion of the evidence that should dic- 
tate cutting off the program entirely or redirecting it 
drastically? 


5. Have economic, demographic, cultural trends been 
explicitly takken into account? How adequately? 


Uncertaint 


Program uncertainty can be of various kinds: e.g., 

re Gan be Uncertainty because of a lack of clarity in 
jectives, risk (statistical uncertainty), new technology. 
actors external to the program (e.g., the future level of 
economic activity), inadequate bases for estimating casts. 
and inadequate bases for estimating consumer demand. 


1. Have the adequacies and inadequacies of data on 
major issues been discussed? Where data are inadequats, 
ew does the PM state what action will be taken to improve the 
situation? 


2. Aside from data problems, are there important 
underlying uncertainties about program effectiveness or 
costs? Is there important technological uncertainty? 
Does the PM state what additional analysis or wsearch is 
being undertaken to reduce these uncertainties? 


3. Have sensitivity testsbeen carried out to deter- 
mine how the uncertainties affect program recommendations: 
4) (e.g., Has a "break even" analysis been made which shows 
what the value of an uncertain parameter would have to be 
in order to get equivalence among alternative programs? 


V. GENERAL 


1. Have the programs covered in the PM been describe: 
Glearly, concisely, and quantitatively? 


2. Are critical assumptions, relationships and facts. 


spelled out? In any program area, a few factors usually 
are much more important than the rest. Does the analysis 
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identify and focus on these critical factors? Or does it 
handle the important ones by assumption and concentrate 
on less important ones? 


3. Have the most important data on a program area 
been presented or are key assertions made without sup- 
porting data? Should the agency be asked to prepare and 
forward additional data prior to the Fall budget review? 


4. Has the program "base" been analyzed or has the 
analysis been limited largely or entirely to new programs? 


5. Have the recommendations presented on a program 
area been compared with earlier agency recommendations on 
the same subject? Have data on actual program performance 
been compared with earlier agency claims for the program? 


6. Is the recommended program composition and level 
based on an overall analysis of {a) need or effective de- 
mand; (b) an arbitrary rule of thunb(e.g. x percent more 
than last year's level); (c) Presidential commitment; 

(d) an estimate of the level that the Congress will accept; 
(e) other considerations? 


7. Where the analysis is inadequate or incomplete, 
does the PM indicate what future work is planned to inm- 
prove it; the data to be gathered, criteria to be applied, 
method of analysis to be used. 


tell _ 
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